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Appendix A. About the study 


This appendix provides an overview of the literature on early indicator and intervention systems (EIISs), including 
key indicators of high school dropout and graduation; which grade levels to focus on, as identified in the research 
for preventing high school dropout; and the evidence on EIISs that was available before this study. It then 
summarizes the history of Oregon Ballot Measure 98 and the High School Success Initiative during its first two 
years to provide additional context. Finally, it offers a more detailed description of each component of an EIIS. 


Review of the literature 

ElISs, also called early warning systems or early warning intervention monitoring systems, are a popular tool used 
by schools and school districts and supported by state education agencies across the country to identify students 
who do not meet on-track thresholds on indicators associated with high school graduation (Allensworth & Easton, 
2007; George Washington University, 2012; Stuit et al., 2016; Therriault et al., 2010; Therriault et al., 2013). 
Students identified as not on track are then assigned to individualized interventions. 


Summary of research on indicators. The most popular early warning indicators (attendance, behavior, and course 
performance) have been thoroughly researched over the past 15 years, resulting in the creation of many related 
data systems and the identification of promising practices for the use of these indicators (Allensworth & Easton, 
2007; Balfanz et al., 2007; Bowers et al., 2013; Bruce et al., 2011; Roderick et al., 2014; Therriault et al., 2013; 
Uekawa et al., 2010). The following discussion reviews some of the key findings around these indicators. In short, 
the evidence on using attendance and course performance data as predictors of dropout and high school 
graduation is strong. The evidence for using behavior indicators as predictors is mixed (Allensworth et al., 2018). 


The Consortium on Chicago School Research at the University of Chicago has been at the forefront of research on 
early warning indicators for many years. One of its early studies identified a key indicator of on-time graduation 
as the combination of accumulating five credits and receiving no more than one failing semester grade in a core 
course (math, English, history, or science) during grade 9. This measure correctly identified students who would 
graduate within four years 80 percent of the time. The study authors also identified grade point average in core 
courses and attendance as effective indicators of graduation, offering further information and additional 
opportunities for identification during grade 9 (Allensworth & Easton, 2007). A recent study by the consortium 
found that similar measures for grade 10 students are also highly predictive of four-year graduation (Seeskin et 
al., 2020). The indicator for on-track graduation at the end of grade 10 is measured as accumulating 13 credits 
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and failing no more than one semester course during grade 10. The consortium also found that attendance and 
grades are more predictive of high school graduation than standardized test scores (Allensworth et al., 2018). 


A study of New York City Department of Education data found similar associations for grade 9 students. The study 
showed that earning 10 or more credits by the end of grade 9 offered a reliable prediction of graduation with a 
New York Regents diploma, which indicates college readiness (Kemple et al., 2013).1 However, the study found 
that passing at least one Regents exam represents a substantial improvement in predicting graduation with a 
Regents diploma.* This combined on-track indicator was reliable and stable across seven cohorts of students. The 
analysis also showed that a substantial increase in grade 9 on-track graduation rates foreshadowed increases in 
Regents diploma graduation rates in New York City. The differences and similarities between the indicators in 
Chicago and New York highlight how local policies can affect indicator choices. 


A study of predictors of dropping out in Delaware found that the strongest indicators in grades 9-12 were 
attendance rates, grade repetition, and math and English language arts course grades (Uekawa et al., 2010). The 
study also found a statistically significant relationship between number of suspensions in grades 9, 10, and 11 and 
dropping out, but the number of suspensions indicator did not demonstrate a statistically significant explanatory 
power when combined with the other indicators. 


A review of 36 studies and 110 possible early warning indicators looked at both the precision (ability to identify 
dropouts) and accuracy (ability to identify nondropouts) of the indicators used (Bowers et al., 2013). The review 
found that longitudinal growth models (looking at indicators such as math achievement trajectories from grades 
7 through 12) were the most precise and accurate, but it acknowledged that these models did not allow for easily 
calculated flags using data readily available at schools. The strongest cross-sectional indicators for which data are 
more likely to be readily available include the freshman on-track indicator created by the Consortium on Chicago 
School Research, mentioned above, along with other course performance measures of cumulative grade point 
average and number of first semester failures. Disciplinary infractions were not found among the strongest 
indicators in this review of the research. 


The Everyone Graduates Center at Johns Hopkins University has worked extensively over the years to help districts 
identify the best indicators of high school dropout and graduation and to support the creation of EllSs. A study 
using longitudinal data demonstrated how predictive indicators reflecting poor attendance, misbehavior, and 
course failures in grade 6 can be used to identify 60 percent of the students who will not graduate from high 
school (Balfanz et al., 2007). Another study found that the number of suspensions in grade 9 was closely linked to 
dropout and high school graduation and that the relationship was statistically significant even after the model 
controlled for student demographic characteristics, attendance, and course performance. The study also found 
evidence that disciplinary infractions can be an early indicator of a downward academic spiral for some students 
and that these behavior issues may appear earlier than attendance and course performance issues (Balfanz et al., 
2014). Still, the use of disciplinary infractions in EllSs should be approached with caution since the research 
findings on these measures are mixed, definitions of disciplinary measures differ across states and districts, and 
male students and students of color tend to have disproportionate suspension rates compared with other 
students who exhibit similar behavior (Allensworth et al., 2018). If these disproportionate rates of disciplinary 
infractions for some groups of students are not proportional to actual student behavior, this would be particularly 
problematic for an EIIS because certain student groups would be identified as at risk and assigned to services at 
higher rates than other groups, regardless of their actual behaviors. 


Focus on grade 9 and 10. The Consortium of Chicago School Research focused on grade 9 course performance 
because many students fall off track during the transition from grade 8 to grade 9. The consortium found that a 


1 Credits are counted by semester in New York and by year in Chicago. 
? Regents exams in several subject areas were required for graduation with a Regents diploma across the state of New York. 
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quarter of students with very high grade 8 standardized test scores were off track (using the on-track for 
graduation in grade 9 measure discussed above) at the end of grade 9, and 40 percent of students with very low 
grade 8 test scores were on track at the end of grade 9 (Allensworth & Easton, 2005). A more recent study found 
that the percentage of students on track at the end of grade 9 improved most when schools acted on real time 
data on absences and course performance and that this improvement led to higher high school graduation rates 
(Roderick et al., 2014). A recent study confirmed that grade 9 is still the most critical year in high school student 
trajectories and that the grade 9 on-track indicator is still highly predictive of graduation. Based on the indicator 
discussed above, nearly 50 percent of the nongraduates studied were off track, and another 20 percent had a 
course failure or low attendance during grade 9. Although the grade 9 indicators were highly predictive of high 
school graduation, the study found that an additional 8 percent of students who were on track in grade 9 moved 
off track in grade 10 and that students’ grades and attendance generally decreased in grade 10 (Seeskin et al., 
2020). 


Summary of evidence for early indicator and intervention systems. The What Works Clearinghouse’s Preventing 
Dropout in Secondary Schools updated practice guide recommends monitoring all students’ progress and 
intervening when students show early signs of attendance, behavior, or academic problems, all the while noting 
that there is minimal evidence for this strategy (Rumberger et al., 2017). There are at least two rigorous studies 
of the implementation and effectiveness of EIISs. 


The most important research on the impact of early warning systems comes from a randomized controlled trial of 
the impacts of the Early Warning Intervention and Monitoring System implemented in the Regional Educational 
Laboratory Midwest region (Faria et al., 2017). The study findings suggest that the system reduced chronic 
absenteeism and course failure, even after its first year and with relatively modest implementation. The study did 
not find a reduction in suspensions or an effect on credit accumulation. A rigorous evaluation of Diplomas Now, a 
full-school reform model that includes an EIIS, did not find impacts on the attendance, behavior, or course 
performance indicators when looked at separately, but did find an impact on a composite indicator of all three 
combined (Corrin et al., 2016). 


The study of the Early Warning Intervention and Monitoring System compared schools implementing the system 
with schools that continued with their normal practices, and the Diplomas Now study compared schools 
implementing Diplomas Now with schools that were either maintaining their normal practices or pursuing other 
types of school reform. The current study of EllSs differed in that it compared districts that adopted EIIS using the 
additional High School Success Initiative funding with districts that might have used the additional funding to 
implement other activities focused on dropout prevention or college or career readiness. 


Summary of Ballot Measure 98 and the High School Success Initiative 


In November 2016 Oregon voters approved Ballot Measure 98 to provide additional funding to school districts for 
high school dropout prevention and college and career readiness initiatives. The measure created the High School 
Success Initiative, which provides direct funding to school districts and charter schools serving high school 
students (grades 9-12) with the goals of improving students’ progress toward graduation starting in grade 9, 
increasing high school graduation rates, and improving the college and career readiness of high school graduates. 
The measure allocated $170 million for the 2017-19 period to be dispersed among districts that serve high school 
students. Funds were granted to districts in three areas: establishing or expanding career and technical education 
programs in high schools, college-level education opportunities for students in high schools, and dropout 
prevention strategies in high schools. More than 250 school districts and charter schools throughout Oregon 
received funding during this first phase of the initiative, including almost all the EllS and comparison districts in 
this study (Oregon Department of Education, 2020a). Equity is a central tenet of the legislation, which specifies 
that traditionally underserved student groups should be targeted for services to ensure that their needs are met 
and that they are able to fully benefit from the initiative (Oregon Department of Education, 2020b). 


REL 2021-097 A-3 


All districts were required to use the funds in at least one of the areas listed above, and some districts, depending 
on the size of the grant, were required to use funds in all three areas. Districts that received less than $100,000 a 
year needed to address at least one of the areas, districts that received $100,000-$350,000 were required to 
address at least two areas, and districts that received $350,000 or more were required to address all three areas 
(Oregon Department of Education, 2020b). All eligible districts received funding in the first year. Prior to the 
second year, eligible districts were required to submit a High School Success Initiative plan describing how they 
intended to use the funding to address any or all of the three areas. All but one of the EIIS districts submitted a 
plan, and all but two comparison districts in the study submitted plans. All district plans were reviewed by the 
study team. Districts with plans that included an EIIS or activities similar to an EllS were removed from the 
comparison pool. 


The initiative specifies the following uses for funding under each of the three areas. For the establishment or 
expansion of career and technical education, funds could be used for the purchase of equipment; the construction 
of facilities; and the recruitment, licensing, employment, and training of personnel to provide career and technical 
education. For the establishment or expansion of college-level opportunities, the funds could be used for 
Advanced Placement, International Baccalaureate, or comparable college-level courses; dual credit and other 
accelerated college credit programs offered in conjunction with a postsecondary education institution; direct 
assistance to students with the selection and completion of any of these college-level opportunities; and the 
recruitment, licensing, employment, and training of personnel to provide college-level education opportunities in 
high school. For the establishment or expansion of dropout prevention initiatives, funds could be used to 
implement activities designed to reduce chronic absenteeism; establish and maintain data management systems 
that provide timely reports on students’ attendance, disciplinary infractions, and course performance; provide 
academic and social supports for students at risk of not graduating, including summer programs, additional 
instructional time before or after school, tutoring or small group instruction during school, or counseling service; 
and provide coaching and counseling on postsecondary education and employment opportunities. The law was 
amended in 2017 to allow some funding (up to 15 percent) to support grade 8 students (Oregon Department of 
Education, 2020b). 


Description of early indicator and intervention systems 

The study team was unable to conduct research on EIIS implementation for this study, but generally in the EIIS 
approach, districts use data systematically to identify students who are not meeting attendance, behavior, or 
course performance goals; assign those students to interventions; and monitor their response to those 
interventions. There are four core elements of EllSs. 


First, the EIIS design calls for a frequently refreshed data system that provides timely information on student 
indicators, often including attendance, behavior, and course performance (Frazelle & Nagel, 2015). Each district 
determines and sets thresholds for each indicator. Guidance on the best indicators suggests that they should be 
predictive of graduation, usable and clear, available in real-time or at the right time, have direct causal linkage to 
graduation or educational attainment, and be malleable enough that support services provided by the schools can 
have an effect (Allensworth et al., 2018). Districts often use indicators that have been shown to be associated with 
the risk of not graduating high school on time (Bowers et al., 2013; Uekawa et al., 2010). Districts can also validate 
their own thresholds using longitudinal local data that measure students’ outcomes across high school, including 
high school graduation (Therriault et al., 2013). A district might set the threshold for attendance so that the system 
identifies any student missing more than 10 percent of school days. A threshold for behavior might be having one 
or more disciplinary infractions. A course performance threshold might be receiving a D or F in one or more 
courses. The data system is refreshed nightly and is available through a web-based dashboard, so district and 
school staff members regularly have access to up-to-date information (Willamette Education Service District, n.d.). 
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Second, the EIIS design offers a menu of interventions available at each school or district to address problems 
related to each indicator. For example, an afterschool tutoring program might be identified as a possible 
intervention for students not meeting course performance goals (Frazelle & Nagel, 2015). The menu is created by 
district or school staff and helps organize the individualized support services available for students. Usually, these 
menus are created in a spreadsheet program that lists interventions as rows and indicators as columns. Some 
interventions can support more than one indicator. For instance, individual coaching can support students 
struggling with attendance, behavior, course performance, or a combination of these issues. Interventions that 
can support more than one indicator can be noted by marking multiple indicator columns in the spreadsheet 
program (Frazelle & Nagel, 2015). 


Third, a school implementing an EIIS creates a response team of teachers and other school staff members who 
meet frequently to discuss data reports from an EIIS and decide which interventions from the menu are best suited 
to each student’s needs. For example, a student not meeting the attendance threshold could be assigned to a 
daily, first-period check-in with a school staff member. Students with disciplinary incidents could be assigned to a 
mediation group where other students trained in conflict resolution help their peers solve disputes. A student 
who falls below the threshold on more than one indicator could be assigned to individual coaching (Frazelle & 
Nagel, 2015). 


Fourth, the response team monitors and updates the support as needed. After assigning a student to a specific 
intervention or set of interventions, the response team monitors the student’s participation and progress to 
determine when changes should be made to the type or strength of the support the student receives (Frazelle & 
Nagel, 2015). 
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Appendix B. Methods 


This appendix discusses the study’s data sources, outcome measures, study sample, and analytic methods and 
provides supplemental information for systematic reviews. 


Data sources and outcome measures 


The main data for this study are student-level records from the Oregon Department of Education for all Oregon 
high school students from 2013/14 to 2018/19. The records data contain information on student attendance, 
disciplinary infractions, course credit accumulation, performances on state standardized tests, and demographic 
characteristics. The study team used these data to assess whether early indicator and intervention systems (EIISs) 
had any first-year effects on chronic absenteeism, disciplinary infractions, course progression, or academic 
performance. The study team examined the outcome measures listed in table B1 in the grade levels identified. 


Table B1. Description of outcome measures 


Outcome measure 


Chronic absenteeism 


The value is 1 if the student was absent 


Chronic absenteeism: percentage of 
students who were absent for more 
than 10 percent of enrolled days 


Severe chronic absenteeism: percentage 


of students who were absent for more 


than 15 percent of enrolled days 


Disciplinary infractions 


Percentage of students who were ever 


suspended or expelled during the school 
year 


year, and 0 otherwise 


Course progression 


The value is 1 if the student obtained at 


Percentage of students who met the 
course-progression goals needed to be 
considered on track to graduate in 
grade 9? 


Academic performance 


Percentage of students who performed 


at or above the proficiency level in 
math? 


Percentage of students who performed 


at or above the proficiency level in 
English language arts” 


a. This measure is available only for students in grade 9. 


Description 


for 10 percent or more of enrolled days, 
and 0 otherwise 


The value is 1 if the student was absent 
for 15 percent or more of enrolled days, 
and 0 otherwise 


The value is 1 if the student was ever 
suspended or expelled during the school 


least 25 percent of the credits required 
for graduation, and 0 otherwise 


The value is 1 if the student’s score on 
the state standardized math test was at 
or above the proficiency level, and 0 
otherwise 


The value is 1 if the student’s score on 
the state standardized English language 
arts test was at or above the proficiency 
level, and O otherwise 


b. This measure is available only for students in grade 11. 


Source: Authors’ summary. 


eT g-Te (= 


All grades (primary analysis); 
by grades 9-12 (exploratory analysis) 


All grades (primary analysis); 
by grades 9-12 (exploratory analysis) 


All grades (primary analysis); 
by grades 9-12 (exploratory analysis) 


Grade 9 (primary analysis) 


Grade 11 (primary analysis) 


Grade 11 (primary analysis) 


There were two measures of chronic absenteeism in this study: chronically absent and severely chronically absent. 
Chronically absent refers to the percentage of students who were absent for more than 10 percent of enrolled 
days. This threshold is suggested by the National High School Center as an early warning indicator for chronic 
absenteeism (Therriault et al., 2013) and is often used by state education agencies (Chang et al., 2019). Severely 
chronically absent refers to the percentage of students who were absent for more than 15 percent of enrolled 
days. This measure is used in this study to capture students with more serious attendance issues (Corrin et al., 
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2016). With this measure the study captures whether EIIS adoption might have led to fewer students being 
severely chronically absent. 


The disciplinary infractions measure refers to the percentage of students who were ever suspended or expelled 
during the year, as reported in the Oregon Department of Education data. Many EIISs and other early warning 
indicator systems track students’ office disciplinary referrals as well as suspensions and expulsions, allowing a 
school’s staff to identify students with behavioral issues early on. The state data used for this study do not include 
office referrals. Although the suspensions and expulsions outcome measure is the best measure of student 
behavior available and has been used in other evaluations of early warning indicator systems (Corrin et al., 2016; 
Faria et al., 2017), it is imperfect because it depends not only on students’ actual behavior but also on choices 
districts and schools make about how to deal with that behavior. For example, in recent years many school districts 
across the country have moved toward using restorative practices, a popular school reform that promotes 
relationship building and mediation over suspension. A district could have fewer suspensions because it adopted 
these or similar practices and not because there were changes in student behavior. It is also possible that staff 
members in EIIS districts might give more disciplinary referrals than they would without an EIIS in order to ensure 
that students receive the EIIS services. 


Academic performance refers to the percentages of grade 11 students who scored proficient or better on the 
Smarter Balanced assessments in math and English language arts. Often, EIIS data systems are set up to identify 
students who have received a D or an F in one or more courses or whose grade point average dips below a certain 
level (such as 2.0 or a C average). These students are then offered interventions such as tutoring, homework help, 
or supplemental course work to strengthen their understanding of the subject matter. This support is theorized 
to lead to stronger course performance and ultimately to better general academic performance. However, data 
were not available to measure students’ course performance directly. The study therefore measured the medium- 
term outcome of student academic performance using grade 11 state standardized test scores in math and English 
language arts. 


The study team also constructed student characteristic measures based on the records data for each baseline year 
and the program year. These measures cover students’ gender, race/ethnicity, special education status, and 
English learner status. The Oregon Department of Education also provided school-level enrollment data for the 
2013/14—2018/19 school years and high school graduation rate data for all baseline years (2013/14—2017/18). 


In addition to student-level records data, the study team collected school-level information from the Common 
Core of Data database maintained by the National Center for Education Statistics (U.S. Department of Education, 
n.d.). These variables include the school’s locality (urban, suburban, town, or rural), eligibility for the federal Title 
| program (which provides additional funds to schools with high percentages or high numbers of children from 
low-income families), and type (charter, magnet, virtual, or regular). 


Data from the EIIS data system managed by the Willamette Education Service District (n.d.) were used to identify 
the program districts. These data were provided to the study team by Willamette Education Service District. No 
other data from the EIIS data system were available to the study team. 


Missing data. The student-level data allowed the team to construct district-level outcome measures for all districts 
in the study sample. Therefore, there was no attrition of districts in any of the main analyses, and there were no 
missing data for any of the outcome measures. This fact, along with the fact that covariates were not used in the 
main analyses (see the analytic methods section below), resulted in there being no missing data in these analyses. 
The attrition of districts for the subgroup analyses is discussed in the study sample section below. 


Study timeframe. One important limitation of this study is that, due to the project timeline, it could assess the 
effect of EllSs only on short-term and some medium-term outcomes for one year. The ElIS model is complex, and 
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it could take more than a year for districts to fully implement all the components effectively. Although the initiative 
would be expected to show some improvements in shorter-term outcomes such as the measures related to 
attendance, behavior, and course progression, it is less likely that the initiative would produce gains on medium- 
term goals such as academic performance (which is measured in this study) and persistence and progression 
through high school (which are not measured because more than one year of data are needed to properly measure 
them). Since it is likely to take more than one year for the initiative to produce an effect on the long-term goal of 
improving high school graduation rates, this study did not explore that outcome. 


Study sample. The primary analysis sample of the study consisted of 65 EIIS districts in Oregon that adopted an 
ENS in 2018/19 and 29 unique comparison districts that are similar to the EIIS districts across a variety of 
characteristics. This section describes the matching process that produced this sample of districts. The same 
process was used to identify the districts included in the subgroup analyses that focused on students not meeting 
attendance, behavior, or course-progression goals in the year prior to EllS adoption. The matching process for the 
subgroups is described at the end of this section. 


The early indicator and interventions systems districts. To be included as an EIIS district in the study, a district had 
to meet several criteria: 


e Use the Oregon Data Suite (a popular ElIS data system) by logging in at least once during the 2018/19 school 
year. 


e Have at least one regular high school (districts that served specific populations or contained only virtual or 
alternative schools were excluded). 


e Have data on all outcome measures for 2013/14—2018/19 in the district records data obtained from the 
Oregon Department of Education. 


This process identified 65 school districts in the East, South, West, and Northwest regions of Oregon that met 
these criteria; there were no EIIS districts in the Central region. These districts represented 97 percent (65 of 67) 
of all districts that logged onto the Oregon Data Suite at least once that year. They are included in the study as 
the EIIS districts. All but one of these districts also reported plans to use the High School Success Initiative funds 
to adopt an EIIS during the 2018/19 school year. These EIIS districts differed from other school districts in Oregon 
in student composition and district structural characteristics. For example, compared with other eligible districts 
in Oregon, the EIIS districts on average had larger student enrollment, higher student-teacher ratios, and higher 
proportions of English learner students. The EIIS districts were similar to other eligible districts in the student 
outcomes that were of interest to this study, as measured in the 2017/18 school year, the year before the EIIS 
was rolled out (table B2). 


1 Although several different data systems are used in Oregon, this study focuses on districts using the Oregon Data Suite, a particularly well- 
developed system and the most popular. 


REL 2021-097 B-3 


Table B2. Outcome measures and student composition and district structural characteristics of early indicator 
and intervention systems (EIISs) districts in the study and all other eligible districts in Oregon in the last 
baseline year, 2017/18 


Oregon Other eligible 
EIIS districts Oregon districts® 
Outcome measure (n =65) (n = 101) 
Percentage of students who were chronically absent, all grades 30.7 30.1 
Percentage of students who were severely chronically absent, all grades 17.4 16.7 
Percentage of students with disciplinary infractions, all grades 7.6 7.0 
Percentage of students meeting course-progression goals, grade 9 80.3 81.5 
Percentage of students proficient or above in math, grade 11 32.1 30.5 
Percentage of students proficient or above in English language arts, grade 11 71.0 69.5 


Student composition 


Percentage of White students 68.9 75.0 
Percentage of male students 51.5 52.3 
Percentage of English learner students 24 ** 1.2 
Percentage of students with Individualized Education Programs 14.2 13.3 


District structural characteristics 


Number of high schools in district 1.4 1.5 
Percentage of high schools located in a city or suburb 22.3 13.1 
Percentage of high schools eligible for Title | 10.7 15.1 
Percentage of high schools serving only grades 9-12 70.1* 52.3 
Percentage of high schools that are charter schools 14.6 19.2 
Average high school enrollment in grades 9-12 687.8*** 360.0 
Average grade 9 enrollment 179.1*** 91.9 
Average grade 10 enrollment 171.9*** 91.6 
Average grade 11 enrollment 167.0*** 88.0 
Student-—teacher ratio 18.9** 16.4 
Average graduation rate (percent) 80.6 79.9 


* Difference significant at p < .05; ** significant at p < .01; *** significant at p < .001. 

Note: Values are observed means aggregated to the district level. A two-tailed t-test was applied to each estimated difference between the EIIS districts and 
other eligible Oregon districts. Rounding might cause slight discrepancies in calculating sums and differences. 

a. Includes only districts that serve regular, nonalternative high schools. 

Source: Authors’ analysis based on student-level data from the Oregon Department of Education and school-level data from U.S. Department of Education 
(n.d.), for school years 2013/14—2017/18. 


Selection of comparison districts. Ina comparative interrupted time series design, the purpose of the comparison 
group is to capture the extent to which the outcomes in program group (EIIS districts) would have changed had 
the districts not adopted the program. The study team used a two-step procedure to maximize the likelihood that 
the difference between the comparison districts’ actual outcomes in the program year and their projected 
outcomes for the program year provided the right counterfactual. First, the study team identified a pool of 
potential comparison districts within the same geographic areas as the EIIS districts. Second, the study team used 


REL 2021-097 B-4 


multiple matching methods to select comparison schools whose baseline characteristics and trends in outcome 
measures were as similar as possible to those of the EIIS districts. 


Identifying the comparison pool. The pool of potential comparison districts consisted of districts that did not plan 
to use their High School Success Initiative funding to adopt an EIIS during 2018/19, the study year. To be part of 
the comparison pool, districts had to meet the following criteria: 


e Had not discussed adopting an early warning system or components of such a system in its 2018/19 High 
School Success Plan.? 


e Was not located in the Central region of Oregon (because none of the EIIS districts were in the Central region). 


e Had at least one regular high school and at least 20 enrolled high school students (districts that served specific 
populations or comprised only virtual or alternative schools were excluded). 


e Had data on all the study outcome measures over 2013/14—2018/19 in the district data obtained from the 
Oregon Department of Education. 


The team identified 75 districts that met these criteria and that were considered potential comparison districts. 


Selecting matched comparison districts. The comparison districts in the study were chosen by matching each of 
the 65 EIIS districts to the one or two districts in the comparison district pool that were the best match. A 
comparison district was considered a best match to an EIIS district if it was the most similar to the EIIS district in 
baseline trends in outcomes and outcome values and in student composition and district structural characteristics 
in the year before an EIIS was adopted. 


Two matching metrics were used to gauge the similarity between an EIIS district and a potential comparison 
district: Euclidean distance and propensity score. The Euclidean distance captures the differences between two 
districts in multiple matching variables. The Euclidean distance across M matching variables is defined as: 


where P,, is the value of variable m for the EIIS district, C,, is the value of variable m for a potential comparison 
district, and SD,, is the standard deviation of variable m across all the EIIS districts and potential comparison 
districts. This distance metric was calculated for each possible pair of EllIS and comparison districts, and the one 
or two comparison districts that were the most similar to (or least distant from) a given EIIS district were identified 
as the matched comparison for that EIIS district. 


The second metric used to measure the overall distance between districts was the propensity score. The 
propensity score is the predicted likelihood (or propensity) of a district being in the EIIS group. This score is 
calculated based on estimated coefficients from the following logistic regression: 

m=M 

logit(Program; ) =at >, bmMmyj + & 

m=1 
where Program, is an indicator that equals 1 if district jis an EIIS district and Oif not, and M,,,; is the m" matching 
variable for district j, a is a constant term, 6,, is the estimated coefficient for the m™ matching variable, and Ejis 
the between-district random variation. Only EIIS districts and potential comparison districts with propensity scores 


? The literature includes studies that used other methods to select comparison districts. One recent example is Athey et al. (2018). 
3 Districts had to submit these plans in the spring of 2018 to receive funding under the High School Success Initiative. 
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within the range of the common support were kept in the matching process and subsequent analysis, to ensure 
that districts with the same M values have a positive probability of being treated and not treated (Heckman et al., 
1999). 


The following variables were used to calculate Euclidean distances and propensity scores in this study: 


e Outcomes for the last baseline year (2017/18) and estimated slopes of baseline trends for each outcome 
measure: 


Percentage of students who were chronically absent (for all grades). 


Oo 


o Percentage of students who were severely chronically absent (for all grades). 
o Percentage of students with disciplinary infractions (for all grades). 


o Percentage of students meeting the course-progression goals to be considered on track to graduate 
(grade 9 only). 


o Percentage of students proficient or above on the state math test (grade 11 only). 

o Percentage of students proficient or above on the state English language arts test (grade 11 only). 
e Student composition in the last baseline year (2017/18): 

o Percentage of White students. 

o Percentage of male students. 


Percentage of English learner students. 


Oo 


Percentage of students with Individualized Education Programs. 


O 


e District structural characteristics in the last baseline year (2017/18): 
o Number of high schools in the district. 
o Percentage of high schools located in a city or suburb. 
© Geographic region (East, South, West, or Northwest). 
o Percentage of high schools eligible for Title |. 
© Percentage of high schools serving only grades 9-12. 
o Percentage of high schools that were charter schools or regular schools. 
o Average high school enrollment in grades 9-12. 
o Student-teacher ratio. 
o Average graduation rate. 


The study team calculated a series of Euclidean distances and propensity scores using different combinations of 
the matching variables (the complete list, outcome values for 2017/18 and their baseline trends, outcome values 
for 2017/18, and trends plus a few variables related to student composition and district structural characteristics)* 
and conducted one-to-one and one-to-two matching using these metrics. Because of the small size of the 
comparison pool, comparison districts were chosen with replacement. In other words, a district from the pool 
could be chosen as the comparison for more than one EIIS district. Though matching with replacement reduces 


4 The demographic variables used here were percentage of White students, percentage of students with individualized education programs, 
percentage of high schools eligible for Title |, percentage of high schools that were charter schools, and the student-teacher ratio. 
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the number of unique comparison districts in the study, it maximizes the similarity of the comparison districts and 
the EIIS districts with respect to the matching variables. This process resulted in a set of matched samples based 
on different combinations of the matching metric (Euclidean distance or propensity score), matching variables, 
and number of matched comparison districts per EIIS district. 


The study team then checked the similarity between the EIIS district and the comparison district for each matched 
sample and identified four matched samples with high levels of baseline similarity between the EIIS district and 
the comparison district. For each of the four samples there was no more than one estimated difference that was 
statistically significant for any of the outcome measures or demographic and structural characteristics in the last 
baseline year, and none of the estimated differences in any variable exceeded 0.25 standard deviation in effect 
size.° 


To further narrow the choices, the study team conducted a validation analysis for each of the four identified 
samples by looking at the estimated effect of an EIIS on the districts in the last baseline year. This effect should be 
zero, because the EIIS districts were not yet rolling out their EllSs. Therefore, this test provides a useful benchmark 
for validating the design and the selection of the comparison districts. This validation test was conducted by 
treating the last baseline year (2017/18) as though it were the program year. Comparison districts were then 
selected again using the shifted baseline period (2013/14—2016/17), and the effects of ElIS were re-estimated. Of 
the four approaches tested, one produced a statistically significant impact on an outcome measure and was 
dropped from the selection. 


Among the remaining three options, the team selected the matched sample that included all EIIS districts and had 
the largest number of unique comparison districts. This approach matched each EIIS district with the most similar 
comparison district (one-to-one match) using the Euclidean distance across the complete set of matching variables 
listed above. This final matched sample consisted of 65 EIIS districts and 29 unique comparison districts. 


There are no statistically significant differences in the outcome measures or in the student composition or district 
structural characteristics between the EIIS districts and their matched comparison districts in the last baseline 
year before the districtwide adoption of ElISs (table B3). Additionally, the magnitudes of the estimated differences 
are small compared with the conventional threshold of caution (0.25 standard deviation in effect size). An omnibus 
test also found no evidence of systematic difference between these two groups (p-value = .25). 


5 This is the criterion used by the What Works Clearinghouse (2020), which is the clearinghouse for education research hosted by the U.S. 
Department of Education’s Institute of Education Sciences. Studies show that quasi-experimental designs are more likely to produce biased 
results when baseline differences exceed this threshold (Ho et al., 2007). 
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Table B3. Outcome measures and student composition and district structural characteristics of early indicator 
and intervention systems (EIISs) districts and comparison districts in the last baseline year, 2017/18 


Simlalerlae| 
EIS Comparison Effect size of error of p-value of 


districts? districts? Estimated estimated estimated estimated 
Baseline measure (n =65) (As) difference difference’ difference difference 


Outcome measure 


Percentage of students who were 30.7 29.9 0.8 0.02 1.64 .625 
chronically absent, all grades 

Percentage of students who were 17.4 16.5 0.9 0.02 1.42 522 
severely chronically absent, all grades 

Percentage of students with disciplinary 7.6 8.2 -0.6 —0.02 0.85 482 
infractions, all grades 

Percentage of students meeting course- 80.3 80.8 -0.6 -0.01 1.85 .765 
progression goals, grade 9 

Percentage of students proficient or 32.1 32.9 -0.9 —0.02 2.58 .742 
above in math, grade 11 

Percentage of students proficient or 71.0 72.4 -1.4 -0.03 2.19 518 


above in English language arts, grade 11 


Student composition 


Percentage of White students 68.9 72.5 -3.6 —0.07 3.10 .250 
Percentage of male students 51.5 51.5 0.0 0.00 0.74 991 
Percentage of English learner students 2.7 1.6 1.0 0.05 0.56 .074 
Percentage of students with Individualized 14.2 13.3 0.9 0.03 0.68 .202 


Education Programs 


District structural characteristics 


Number of high schools in the district 1.4 1.3 0.1 0.14 0.18 .450 
Percentage of high schools located in a city 22.3 16.9 5.4 0.12 8.11 508 
or suburb 

Percentage of high schools eligible for Title | 10.7 6.2 4.5 0.14 5.58 418 
Percentage of high schools serving only 70.1 71.0 -1.0 -0.02 9.14 .916 
grades 9-12 

Percentage of high schools that are charter 14.6 13.3 1.3 0.03 6.52 845 
schools 

Average high school enrollment in grades 687.8 621.1 66.8 0.10 128.90 .606 
9-12 

Average grade 9 enrollment 179.1 159.5 19.6 0.11 33.94 565 
Average grade 10 enrollment 171.9 159.9 12.0 0.07 32.92 .716 
Average grade 11 enrollment 167.0 149.3 17.6 0.10 30.88 569 
Student-to-teacher ratio 18.9 18.9 0.0 0.01 0.75 975 
Average graduation rate (percent) 80.6 82.5 -1.9 —0.13 1.95 321 


Note: No differences were significant at p = .05 or higher. All values are aggregated to the district level. Student-level standard deviations were used for 
outcome measures and student composition; school-level standard deviations were used for district structural characteristics except for number of high 
schools in district, for which district-level standard deviation was used. A two-tailed t-test was applied to each estimated difference between the EIIS districts 
and the comparison districts. An omnibus test for systematic difference between the EIIS and comparison districts yielded a p-value of .250. Rounding might 
cause slight discrepancies in calculating sums and differences. 

a. The values are the weighted means across districts, where the weights account for the fact that some comparison districts are chosen as the match for 
more than one EIIS district. 

b. Calculated as the proportion of the standard deviation for a given variable. 

Source: Authors’ analysis based on student-level data from the Oregon Department of Education and school-level data from U.S. Department of Education 
(n.d.), for school years 2013/14—2017/18. 
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Selection of the comparison districts for subgroups. A similar matching process was used to select comparison 
districts for subgroups of students: students who were severely chronically absent, who had disciplinary 
infractions, or who had not accumulated enough credits by the end of grade 9 to be considered on track to 
graduate. To explore whether the adoption of an EIIS benefited students in these subgroups, the study team 
constructed three subgroups for the year before EIIS adoption based on measures similar to EIIS indicators (see 
appendix C for more information about the subgroups). Once each subgroup was identified, the team generated 
aggregated district-level outcome measures and student composition and district structural characteristic 
measures based on data only for the specific subgroup of students. The same matching approach as described 
above for the full sample was then used for each subgroup of students; the process resulted in the creation of 
different samples of EIIS districts and comparison districts for each subgroup.°® 


EIS districts with small enrollment numbers or high mobility might have had no students that met the subgroup 
criteria in a given year. When that was the case for either the last baseline year or the program year, the district 
was dropped from the subgroup analysis because no outcome data were available for the matching or for the 
impact analysis. As a result, each subgroup sample included a different number of EIIS districts: there were 46 EIIS 
districts in the subgroup of severely chronically absent students, 50 in the subgroup of students with disciplinary 
infractions, and 60 in the subgroup of grade 10 students who, by the end of grade 9, had not attained the credits 
needed to be considered on track to graduate. Districts with no students meeting the subgroup criteria in 2017/18 
or 2018/19 were excluded from the comparison pool for the same reason. 


This matching process also led to a separate set of comparison districts for each subgroup. The selected 
comparison districts offered the best match to the EIIS districts in terms of outcome measures and student 
composition and district structural characteristics in the last baseline year (2017/18), and the maximum number 
of unique districts were included in the sample. Even more important, for each of the three subgroups there were 
no statistically significant differences between EIIS and comparison districts in outcome measures, student 
composition, and district structural characteristics during the last baseline year (tables B4—B6). Outcome 
measures and student composition and district structural characteristics were measured for the groups of 
students meeting the definition for each of the three subgroups based on their performance during the previous 
year (2016/17). 


Analytic methods 


Comparative interrupted time series. This study evaluated the early effects of EllSs on student outcomes using a 
comparative interrupted time series (CITS) design, a quasi-experimental design that lends itself well to exploring 
the effect of systemwide interventions. In education, CITS designs have been used to evaluate federal policies 
such as No Child Left Behind and the effect of whole-school reform models (Dee & Jacob, 2011; Kemple et al., 
2005; Somers & Garcia, 2016; Somers & Haider, 2017; Wong et al., 2009). Studies have shown that a well- 
implemented CITS design can, in some circumstances, reproduce the results of a randomized experiment 
(Fretheim et al., 2013; Somers et al., 2013; St. Clair et al., 2014). 


®The matching variables used for selecting comparison districts for the subgroup analysis were the same as those used for the full-sample 
matching except that for the grade 10 subgroup of students not obtaining the credits needed to be considered on track in grade 9, outcomes 
based on information from other grade levels were not used as matching variables, and student-composition variables were calculated 
using grade 10 information only. 
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Table B4. Outcome measures, student composition, and district structural characteristics of early indicator 
and intervention systems (EIISs) districts and comparison districts in the last baseline year (2017/18) for 
students identified as severely chronically absent in the previous year (2016/17) 


S\r-laler-lae| 
EIIS Comparison Effect size of error of p-value of 


districts® districts? Estimated estimated estimated estimated 
Baseline measure (n = 46) (ie) difference difference’ difference difference 


Outcome measure 


Percentage of students who were 84.5 84.8 -0.4 -0.01 2.19 .867 
chronically absent, all grades 

Percentage of students who were severely 67.8 70.1 —2.3 —0.06 2.93 443 
chronically absent, all grades 

Percentage of students with disciplinary 15.0 14.6 0.4 0.01 1.99 .852 
infractions, all grades 

Percentage of students meeting course- 42.1 42.9 -0.8 —0.02 5.41 885 
progression goals, grade 9 

Percentage of students proficient or above 9.7 111 -1.4 -0.03 2.78 .612 
in math, grade 11 

Percentage of students proficient or above 52.5 50.0 2.6 0.06 5.71 .655 


in English language arts, grade 11 
Student composition 


Percentage of White students 68.7 71.3 —2.6 0.05 3.66 .476 
Percentage of male students 47.2 49.8 —2.6 -0.05 1.90 173 
Percentage of English learner students 2.6 2.3 0.3 0.02 0.83 .726 
Percentage of students with Individualized 21.4 20.2 1.1 0.03 2.40 .641 


Education Programs 
District structural characteristics 


Number of high schools in the district 1.5 1.5 0.0 0.02 0.28 .938 
Percentage of high schools located in a city 20.7 19.6 1.1 0.02 9.71 911 
or suburb 

Percentage of high schools eligible for Title | 6.4 9.0 —2.6 —0.08 6.11 .673 
Percentage of high schools serving only 73.4 66.4 7.1 0.15 10.29 493 
grades 9-12 

Percentage of high schools that are charter 12.0 15.4 -3.5 -0.09 6.75 .609 
schools 

Average high school enrollment in 702.3 715.2 -12.9 —0.02 139.01 .926 
grades 9-12 

Average grade 9 enrollment 180.0 183.6 -3.6 0.02 36.05 921 
Average grade 10 enrollment 176.3 184.9 -8.5 —0.05 35.82 .812 
Average grade 11 enrollment 172.1 169.8 2.3 0.01 33.33 944 
Student—teacher ratio 19.6 19.2 0.4 0.10 0.82 .603 
Average graduation rate (percent) 80.0 80.7 -0.8 —0.05 2.46 .753 


Note: No differences were significant at p = .05 or higher. All values are aggregated to the district level. Outcome measures and student composition values 
were measured for the subgroup of students who were severely chronically absent (absent for more than 15 percent of school days) in the previous school 
year (2016/17). Only districts with at least one student identified for this subgroup in the program year (2018/19) and the last baseline year (2017/18) were 
included in the analysis. Student-level standard deviations were used for outcome measures and student composition; school-level standard deviations were 
used for district structural characteristics except for number of high schools in district, for which district-level standard deviation was used. A two-tailed t- 
test was applied to each estimated difference between the EIIS districts and the comparison districts. An omnibus test for systematic difference between 
the EIIS and comparison districts yielded a p-value of .783. Rounding might cause slight discrepancies in calculating sums and differences. 

a. The values are the weighted means across districts, where the weights account for the fact that some comparison districts are chosen as the match for 
more than one EIIS district. 

b. Calculated as the proportion of the standard deviation for a given variable. 

Source: Authors’ analysis based on student-level data from the Oregon Department of Education and school-level data from U.S. Department of Education 
(n.d.), for school years 2013/14—2017/18. 
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Table B5. Outcome measures, student composition, and district structural characteristics of early indicator 
and intervention systems (EIISs) districts and comparison districts in the last baseline year (2017/18) for 
students with disciplinary infractions in the previous year (2016/17) 


Sirlalerlae| 
EIS Comparison Effect size of — error of p-value of 


districts? districts? Estimated estimated estimated estimated 
Baseline measure (710) (Xe) difference difference’ difference difference 


Outcome measure 


Percentage of students who were 49.8 53.4 -3.6 —0.08 3.00 .236 
chronically absent, all grades 

Percentage of students who were severely 35.7 36.7 -1.1 —0.03 3.13 729 
chronically absent, all grades 

Percentage of students with disciplinary 32.5 35.7 —3.2 -0.12 2.82 .265 
infractions, all grades 

Percentage of students meeting course- 53.7 54.2 -0.5 -0.01 4.00 .892 
progression goals, grade 9 

Percentage of students proficient or above 9.6 8.2 1.4 0.03 2.55 587 
in math, grade 11 

Percentage of students proficient or above 44.2 39.9 4.3 0.10 4.70 365 


in English language arts, grade 11 
Student composition 


Percentage of White students 61.6 65.0 -3.4 —0.07 4.33 432 
Percentage of male students 69.8 71.1 -1.4 -0.03 1.89 463 
Percentage of English learner students 4.0 3.6 0.4 0.02 1.19 .726 
Percentage of students with Individualized 26.5 22.6 3.9 0.12 2.11 .068 


Education Programs 
District structural characteristics 


Number of high schools in the district 1.5 1.7 -0.1 -0.12 0.26 .641 
Percentage of high schools located in a city 29.0 27.0 2.0 0.04 10.17 845 
or suburb 

Percentage of high schools eligible for Title | 5.9 0.3 5.6 0.17 3.41 .103 
Percentage of high schools serving only 82.1 76.0 6.0 0.13 8.23 .467 
grades 9-12 

Percentage of high schools that are charter 8.0 17.2 -9.2 —0.24 5.27 .085 
schools 

Average high school enrollment in 862.2 789.9 72.3 0.10 142.36 .613 
grades 9-12 

Average grade 9 enrollment 224.6 198.7 25.9 0.14 38.02 498 
Average grade 10 enrollment 215.9 197.7 18.3 0.10 36.16 .615 
Average grade 11 enrollment 208.6 193.8 14.9 0.09 34.28 .665 
Student-—teacher ratio 20.4 20.2 0.1 0.03 0.69 .850 
Average graduation rate (percent) 79.9 76.9 3.0 0.20 2.29 195 


Note: No differences were significant at p = .05 or higher. All values are aggregated to the district level. Outcome measures and student composition values 
were measured for the subgroup of students who had at least one disciplinary infraction in the previous school year (2016/17). Only districts with at least 
one student identified for this subgroup in the program year (2018/19) and the last baseline year (2017/18) were included in the analysis. Student-level 
standard deviations were used for outcome measures and student composition; school-level standard deviations were used for district structural 
characteristics except for number of high schools in district, for which district-level standard deviation was used. A two-tailed t-test was applied to each 
estimated difference between the EIIS districts and the comparison districts. An omnibus test for systematic difference between the EIIS and comparison 
districts yielded a p-value of .273. Rounding might cause slight discrepancies in calculating sums and differences. 

a. The values are the weighted means across districts, where the weights account for the fact that some comparison districts are chosen as the match for 
more than one EIIS district. 

b. Calculated as the proportion of the standard deviation for a given variable. 

Source: Authors’ analysis based on student-level data from the Oregon Department of Education and school-level data from U.S. Department of Education 
(n.d.), for school years 2013/14—2017/18. 
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Table B6. Outcome measures, student composition, and district structural characteristics of early indicator 
and intervention systems (EIISs) districts and comparison districts in the last baseline year (2017/18) for grade 
10 students who had not in the previous year obtained 25 percent of the credits needed to graduate 
(2016/17) 


Sir-lalerlae| 
EIS Comparison ai -Yoi mk) 74-M 0) a =1 000) 0) p-value of 


districts? districts? Estimated estimated estimated estimated 
Baseline measure (7610) (n = 34) difference difference’ difference difference 


Outcome measure 


Percentage of students who were 56.6 56.4 0.2 0.00 3.74 .960 
chronically absent, grade 10 

Percentage of students who were 41.8 40.6 1.3 0.03 3.71 729 
severely chronically absent, grade 10 

Percentage of students with disciplinary 19.8 19.9 0.0 0.00 2.78 991 


infractions, grade 10 


Student composition 


Percentage of White students 62.5 68.6 -6.1 0.13 4.59 .188 
Percentage of male students 60.6 59.7 0.9 0.02 2.75 735 
Percentage of English learner students 5.7 4.2 1.5 0.08 1.73 391 
Percentage of students with Individualized 25.3 23.3 2.0 0.06 2.74 471 


Education Programs 


District structural characteristics 


Number of high schools in the district 1.5 1.5 0.0 0.00 0.24 1.000 
Percentage of high schools located in a city 24.2 22.5 1.7 0.04 8.71 .849 
or suburb 

Percentage of high schools eligible for Title | 8.3 2.1 6.1 0.18 4.22 .151 
Percentage of high schools serving only 74.2 78.4 4,2 -0.09 8.27 .614 
grades 9-12 

Percentage of high schools that are charter 14.2 12.0 2.2 0.06 6.02 .718 
schools 

Average grade 10 enrollment 184.2 174.4 9.9 0.06 33.63 .770 
Student-—teacher ratio 19.4 19.1 0.3 0.08 0.67 .626 
Average graduation rate (percent) 80.0 79.3 0.7 0.05 2.12 727 


Note: No differences were significant at p = .05 or higher. All values are aggregated to the district level. Outcome measures and student composition values 
were measured for the subgroup of grade 10 students who failed to obtain at least 25 percent of the credits required for high school graduation at the end 
of grade 9 in the previous year (2016/17). Only districts with at least one student identified for this subgroup in the program year (2018/19) and the last 
baseline year (2017/18) were included in the analysis. Student-level standard deviations were used for outcome measures and student composition; school- 
level standard deviations were used for district structural characteristics except for number of high schools in district, for which district-level standard 
deviation was used. A two-tailed t-test was applied to each estimated difference between the EIIS districts and the comparison districts. An omnibus test for 
systematic difference between the EIIS and comparison districts yielded a p-value of .898. Rounding might cause slight discrepancies in calculating sums and 
differences. 

a. The values are the weighted means across districts, where the weights account for the fact that some comparison districts are chosen as the match for 
more than one EIIS district. 

b. Calculated as the proportion of the standard deviation for a given variable. 

Source: Authors’ analysis based on student-level data from the Oregon Department of Education and school-level data from U.S. Department of Education 
(n.d.), for school years 2013/14—2017/18. 


In a CITS design, program impacts are evaluated by looking at whether school districts that implemented a 
program (in this study, an EIIS) deviated from their baseline trends by a greater amount than a group of similar 
comparison districts. The following steps outline how the analysis was carried out in the current study: 
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1. Calculating mean outcomes. Student-level data were used to create a time series of an outcome measure 
by calculating the mean outcome for the EIIS districts and for the comparison districts in each year 
examined in the study. The school years before adoption of an EIIS (2013/14—2017/18) are referred to as 
the “baseline years,” and the year in which an EIIS was adopted in the EIIS districts (2018/19) is the 
“program year.” 


2. Estimating baseline trends. The time series of each outcome measure was then used to estimate a trend 
in outcomes during the baseline years. Baseline trends were estimated separately for the EIIS districts and 
for the comparison districts. These trends were then projected into the program year, to determine what 
the outcome would have been if the baseline trends continued. 


3. Obtaining deviations from the baseline trend. The actual program-year mean outcome was then compared 
with the outcome level projected by the baseline trend. The differences between the actual mean 
outcomes and the outcome levels predicted by the trends—the deviations from the baseline trend—were 
estimated separately for the EIIS districts and for the comparison districts. 


4. Estimating the effect of the EllS. The effect of the EIIS was estimated as the difference between the 
average deviation from the trend for the EIIS districts and the average deviation from the trend for the 
comparison districts. If the EIIS model was more effective than the other programs or reforms available 
to districts (those used by the comparison districts), the EIIS districts should experience a deviation from 
their baseline trend that is larger than the deviation found for the comparison districts. 


In this design the effect of the EIIS is not estimated by directly comparing the actual outcomes of the EIIS and the 
actual outcomes of the comparison districts but by comparing the amount by which the two groups deviated from 
their baseline trends. This feature illustrates an important assumption of the CITS design: the comparison districts’ 
deviation from their baseline trend is considered as a valid counterfactual. In other words, the comparison 
districts’ deviation from baseline trend represents the deviation from trend that the EIIS districts would have 
experienced had they not adopted the EIIS. To maximize the likelihood that this assumption holds, the study team 
matched the EIIS districts with comparison districts with the most similar outcomes, their corresponding trends, 
and characteristics in the baseline period. Matching on the outcomes and their baseline trends makes it more 
probable that the EIIS and comparison districts were motivated and affected by the same contextual factors. 


Analytically, the following two-level model was used to estimate the effect of the EIIS in the program year. For the 
analysis all student-level data were aggregated to the district level by year. A multilevel model with district-by- 
year observations clustered within districts was used to account for the clustering of time within districts. 


Level 1. District-years within districts 

Yit = Qoj + BoPROGRAM; + hojRELYR, + AgRELYR, * PROGRAM, + a,POST, + ByPROGRAM; * POST, 
+ Ejt 

Level 2. Districts 


Aj =A + Mj 
oj =hot GF 


where j denotes districts and t is the time index, which is equal to 1 for the first program year (the 2018/19 school 
year) and to O or negative for the five baseline years (0 = the 2017/18 school year, —1 = the 2016/17 school year, 
and so on). 


The variables in the model are defined as follows: 
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Yie = Outcome for district j and year t. 


RELYR; = Continuous variable for time centered at the last baseline 
(pre-intervention) year, such that RELYR = 0 for the 2017/18 
school year (the year before EIIS adoption). 


PROGRAM; = Dichotomous indicator for whether district j is an EIIS 
district and is equal to 1 if an EIIS districts or to O if a 
comparison district. 


POST, = Dichotomous indicator for the program year and is equal to 
1 if the 2018/19 school year or to 0 if a baseline year. 


Ejt = Random variation in outcome across year t within district j. 
Hj =  Between-district random variation in baseline intercept. 
0; = _ Between-district random variation in baseline slope. 


From the model the following quantities of interest can be obtained: 


Ao = Baseline mean (intercept) for the comparison districts in the last 
baseline year. 


ay + Bo = Baseline mean (intercept) for the EIIS districts in the last baseline 
year. 
Po = _ Baseline slope for the comparison districts. 
do + Ao = Baseline slope for the EIIS districts. 
ay = Deviation from baseline trend for the comparison districts in the 


program year. 


a,+ Bp, = Deviation from baseline trend for the EIIS districts in the program 
year. 
By = Estimated effect of the EIIS on Y in the program year. 


In this model, 8, represents the estimated effect of the EllS—that is, the deviation from baseline trend for the EIIS 
districts minus the deviation from trend for the comparison districts. The analysis is weighted to account for the 
fact that some comparison districts are chosen as the match for more than one EIIS district.’ 


The What Works Clearinghouse (2020) standards (version 4.1) also call for statistical adjustments for any 
characteristics with baseline differences that are larger than 0.05 standard deviation in effect size. Table B3 shows 
that no such student-level characteristics were found for the current sample. Therefore, no statistical adjustment 
was included in the primary impact estimation model. The study team conducted a sensitivity test that included 
adjustments for all characteristics with baseline differences larger than 0.05 standard deviation in effect size. 
Results for that test were not substantively different from those based on the primary estimation model. Those 
results are presented in table B7 (column 3) and discussed in the section below on sensitivity tests. 


7 The weights are normalized so that their sum equals the number of districts in the analysis. 
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The same analytic approach was used to assess the effects of EllIS adoption for students across all high school 
grades, as well as to explore such effects by grade levels whenever possible. 


Sensitivity tests. The study team conducted a series of sensitivity tests to check the validity and credibility of the 
design and of the comparison group (see table B7). The impact estimates discussed in the main report are 
presented in the first column of table B7 for reference purposes. 


The study team conducted a design validation test by estimating the “effects” of the EIIS for the last baseline year 
before it was adopted. Since the effects of the program are expected to be zero in the last baseline year, this test 
provides a useful benchmark against which to check the choice of comparison districts. In other words, if the 
estimated effects of the EIIS in the last baseline year are not zero, it would cast doubt on the credibility of the 
selected comparison districts and of the findings. The findings from the design validation analysis are presented 
in the second column of table B7. As expected, they are not different from zero to a statistically significant degree. 


Another threat to the credibility of the impact findings is that if the EIIS districts and the comparison districts 
experienced different demographic changes in the program year, the comparison districts might not serve as valid 
counterfactuals for the EIIS districts. To assess this possibility, the study team re-estimated the effects of EIlSs, 
controlling for student composition and district structural characteristics in the regression model. In general, 
adjusting for student composition and district structural characteristics does not change the size or the statistical 
significance of the estimated effects of EllSs on any of the outcomes (see column 3 in table B7). This result indicates 
that the findings presented in the report are not likely to be confounded with differential shifts in the student 
composition or district structural characteristics in either the EIIS or the comparison districts. 


The study team also re-estimated the effects of EllSs using a set of comparison districts selected through an 
alternative matching metric. In place of Euclidean distance, comparison schools were selected using propensity 
score matching (see the study sample section for a description of this metric) based on outcomes and trends plus 
a few demographic variables, as described earlier, and using one-to-two matching with replacement. To ensure 
that the remaining districts were comparable, this alternative matching approach excluded EIIS districts or 
comparison districts with estimated propensity scores that were outside the range of propensity score values 
shared by these two groups of districts (Caliendo & Kopeinig, 2005). This approach was chosen from all propensity 
score matching approaches because it produced a set of comparison districts that were the most like the EIIS 
districts on average. However, this approach was only able to find suitable matches for 51 of the 65 EIIS districts, 
so the findings might not be directly comparable to the findings from the full sample. In addition, the smaller 
sample size makes it harder to detect statistically significant effects. These two factors might explain why some of 
the findings reported in the last column of table B7 are different from the findings in the main report. 
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Table B7. Sensitivity tests for the estimated effects of early indicator and intervention systems (EIISs) on 
chronic absenteeism, disciplinary infractions, course progression, and academic achievement in 2018/19 


Impact 
findings 
presented Alternative Alternative 

Tamaatlia Design maakeye(=1| matching 
Outcome report validation? specification’ approach‘ 
Percentage of students who were chronically absent, all grades —3.9* -0.9 -3.9* -1.3 
Percentage of students who were severely chronically absent, —3,3** -0.8 —3.4** -0.6 
all grades 
Percentage of students with disciplinary infractions, all grades 0.3 -0.7 0.4 1.8* 
Percentage of students meeting course-progression goals, grade 9 2.4 2.6 2.8 1.8 
Percentage of students proficient or above in math, grade 11 4.9 3.4 5.6 4.0 
Percentage of students proficient or above in English language arts, -0.2 -0.1 0.4 0.1 
grade 11 
Number of EIIS districts 65 65 65 51 
Number of unique comparison districts 29 33 29 39 


* Significant at p < .05; ** significant at p < .01. 

Note: All values are aggregated to the district level. The values in this table are the estimated effects of EIIS, calculated as the differences between the EIIS 
districts and the comparison districts with respect to the deviations from their baseline trends. A two-tailed t-test was applied to each estimated difference. 
a. The validity of the study design was tested by estimating the “effects” of EIIS in the last baseline year before the adoption of EIIS, using the same process 
for comparison district selection as the one used for the main study sample. If the study design and the comparison district selection approach are sound, 
then the “effects” of ElIS in the year before it launched should all be zero by definition. 

b. This specification controlled for certain student composition and district structural characteristics: percentage of White students, the percentage of male 
students, percentage of English learner students, percentage of students with Individualized Education Programs, percentage of schools eligible for Title |, 
average school enrollment, and average student-teacher ratio. 

c. This validation test used propensity scores rather than Euclidean distances to select comparison districts. For each EIIS district the two comparison districts 
with the closest propensity scores were selected and included in the analysis. 

Source: Authors’ analysis based on student-level data from the Oregon Department of Education and school-level data from U.S. Department of Education 
(n.d.), for school years 2013/14—2017/18. 


Supplemental information for systematic reviews 

Standard deviations for the outcomes, student composition, and district structural characteristics measured in the 
last baseline year are in table B8. The standard deviations for student outcomes and composition were calculated 
using individual student data and are thus student-level standard deviations. District structural characteristics 
were measured for each district based on school-level data, and their standard deviations are at the school level, 
with one exception. The number of high schools in a district measure was available only at the district level, and 
therefore the standard deviation for this variable was calculated at the district level. Standard deviations are 
shown for the EIIS districts and comparison districts separately and for both groups of districts pooled together. 


In this study the minimum detectable effect is the smallest true program impact that can be detected with 80 
percent power for a given level of statistical significance (in this case, 5 percent for a two-tailed t-test). Table B9 
presents the realized minimum detectable effects for the study sample and the three subgroups. The minimum 
detectable effects were calculated by multiplying the standard error of each estimated impact by 2.8. 
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Table B8. Standard deviations of student composition and district structural characteristics in the last baseline 
year (2017/18) for early indicator and intervention systems (EIISs) districts, comparison districts, and all 
districts 


EIS Comparison All 


districts districts districts? 
Baseline measure (7iao}s)) (a) (n = 94) 


Outcome measure 


Percentage of students who were chronically absent, all grades 47.0 46.4 46.9 
Percentage of students who were severely chronically absent, all grades 40.6 38.9 40.2 
Percentage of students with disciplinary infractions, all grades 27.6 26.8 27.4 
Percentage of students meeting course-progression goals, grade 9 38.6 39.9 38.9 
Percentage of students proficient or above in math, grade 11 47.3 48.5 47.6 
Percentage of students proficient or above in English language arts, grade 11 45.3 43.8 45.0 


Student composition 


Percentage of White students 49.2 44.9 48.5 
Percentage of male students 50.0 50.0 50.0 
Percentage of English learner students 21.1 13.4 19.4 
Percentage of students with Individualized Education Programs 33.7 33.0 33.5 


District structural characteristics 


Number of high schools in the district 1.1 0.6 1.0 
Percentage of high schools located in a city or suburb 45.1 44.2 44.7 
Percentage of high schools eligible for Title | 36.0 27.0 33.6 
Percentage of high schools serving only grades 9-12 46.6 49.8 47.6 
Percentage of high schools that are charter schools 36.0 42.7 38.1 
Average high school enrollment in grades 9-12 746.2 583.5 702.0 
Average grade 9 enrollment 198.2 155.4 186.8 
Average grade 10 enrollment 187.4 148.2 176.7 
Average grade 11 enrollment 179.5 142.7 169.4 
Student-teacher ratio 4.5 4.0 4.4 
Average graduation rate (percent) 14.1 16.2 14.7 


Note: Standard deviations for outcome measures and student composition variables were calculated at the student level. Standard deviations for number 
of high schools in the district were calculated at the district level. Standard deviations for the rest of the district structural characteristics variables are 
calculated at the school level. 

a. These values were used for baseline-year calculations of effect sizes. 

Source: Authors’ analysis based on student-level data from the Oregon Department of Education and school-level data from U.S. Department of Education 
(n.d.), for school years 2013/14—2017/18. 
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Table B9. Minimum detectable effects of early indicator and intervention systems for full study sample and 
the three subgroup samples, by outcome measure and sample, 2018/19 


SU] of-4go}0] omcy-1anle) (=) 


Grade 9 

Full Chronic Disciplinary on-track 
Outcome measure sample absenteeism? infractions? status‘ 
Percentage of students who were chronically absent, all grades 4.4 10.5 9.7 14.9 
Percentage of students who were severely chronically absent, 3.1 12.1 9.4 14.1 
all grades 
Percentage of students with disciplinary infractions, all grades 1.8 6.2 7.8 7.3 
Percentage of students meeting course-progression goals, grade 9 6.2 20.6 17.3 na 
Percentage of students proficient or above in math, grade 11 11.9 13.0 12.2 na 
Percentage of students proficient or above in English language arts, 7.9 22.8 18.9 na 
grade 11 
Number of early indicator and intervention system districts 65 46 50 60 
Number of unique comparison districts 29 23 26 34 


na is not applicable because grade 9 on-track status does not pertain to other grades. 

Note: All values are aggregated to the district level. Minimum detectable effects in this table were calculated by multiplying the standard error of the 
estimated effects by 2.8, assuming a statistical significance level of 5 percent. 

a. This subgroup includes students who were absent for more than 15 percent of school days in the 2017/18 school year. 

b. This subgroup includes students who were ever suspended or expelled during the 2017/18 school year. 

c. This subgroup includes grade 10 students who failed to obtain at least 25 percent of the credits required for high school graduation by the end of grade 9. 
Source: Authors’ analysis based on student-level data from the Oregon Department of Education and school-level data from U.S. Department of Education 
(n.d.), for school years 2013/14—2017/18. 
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Appendix C. Supporting analyses 

This appendix supplements the findings on the effects of early indicator and intervention systems (EIISs) presented 
in the main report. It includes additional details on the full sample findings presented in the report, as well as 
findings for the student subgroup analyses. 


Additional details on the primary findings in the report 

Figure 2 and tables 1—4 in the main report present the findings on the first-year effects of EllSs on the percentages 
of students who were chronically absent, had disciplinary infractions, met the course-progression goals by the end 
of grade 9 needed to be considered on track to graduation, and achieved proficiency on state standardized math 
and English language arts tests in grade 11. The details of these impact estimates—effect sizes, standard errors, 
confidence intervals, and p-values associated with each estimate—are presented in table C1. 
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Table C1. Estimated effects of early indicator and intervention systems (EIISs) on chronic absenteeism, 
disciplinary infractions, course progression, and academic achievement for full sample and subsamples, 
2018/19 


Difference between 
actual and projected 

outcomes 95% confidence interval 
(percentage points) 


(percentage points) 


Estimated 

EIS Comparison effect? p-value for 

districts? districts (percentage Lower Upper Siilarel-lac! estimated 
Outcome (n =65) (ais)) rexeyfaiasy) bound bound Effect size error effect 
All grades —3.5** 0.3 —3.9* -7.0 -0.8 0.08 1.58 .014 
Grade 9 —2.2 1.9 -0.3 —4.2 3.6 —0.01 1.98 .869 
Grade 10 -2.9 2.2 —5.1* —9.3 -0.9 —0.11 2.14 .018 
Grade 11 —5.2** 1.3 —6.4** -10.9 —2.0 —0.14 2.25 .004 
Grade 12 3.3 1.1 -4.4 —10.1 1.3 —0.09 2.89 .128 


Percentage of students who were severely chronically absent 


All grades —2.5** 0.8 -3.3** —5.5 -1.1 —0.09 1.12 .003 
Grade 9 -1.5 -1.1 -0.4 -3.0 2.2 -0.01 1.31 759 
Grade 10 -1.9 3.2** —5.2** -8.3 —2.0 —0.14 1.61 .001 
Grade 11 —3.8** 1.8 —5.6*** -8.9 -2.4 —0.14 1.66 .001 
Grade 12 —2.2 -0.1 —2.1 -6.5 2.3 —0.05 2.23 355 
Percentage of students with disciplinary infractions 

All grades 0.3 0.0 0.3 -0.9 1.6 0.02 0.63 591 
Grade 9 0.6 —2.1* 2.8* 0.2 5.3 0.11 1.30 .032 
Grade 10 1.2 1.2 0.0 -1.9 1.8 0.00 0.94 991 
Grade 11 -0.3 0.3 —0.6 —2.8 1.6 —0.03 1.12 582 
Grade 12 -0.2 0.3 -0.6 —2.4 1.3 —0.03 0.95 552 


Percentage of students meeting course-progression goals 

Grade 9 0.7 -1.7 2.4 —2.0 6.8 0.06 2.23 .286 
Percentage of students proficient or above in math 

Grade 11 12.7*** 7.8* 4.9 -3.5 13.3 0.10 4.26 .252 
Percentage of students proficient or above in English language arts 

Grade 11 2.7 3.0 —0.2 5.8 5.3 -0.01 2.81 .930 


* Significant at p < .05; ** significant at p < .01; *** significant at p < .001. 

Note: All values were aggregated to the district level. A two-tailed t-test was applied to each estimated difference between the EIIS districts and the 
comparison districts. Rounding might cause slight discrepancies in calculating the sums and differences. 

a. Values are the estimated differences between actual and projected outcomes for districts that adopted an EIIS in 2018/19. 

b. Values are the differences between the EIIS districts and the comparison districts in the differences between their actual and projected outcomes. 
Source: Authors’ analysis based on student-level data from the Oregon Department of Education and school-level data from U.S. Department of Education 
(n.d.), for school years 2013/14—2018/19. 


Subgroup findings 


Subgroup definitions. \deally, the study team would be able to identify the subgroup of students who were 
targeted by an EIIS and explore the effects of an ElIS on these targeted students. However, such data were not 
available. Instead, the study team used available student-level records from the Oregon Department of Education 


for the school year before EIIS adoption (2017/18) to identify three subsets of students who might have been 
identified for additional services if an ElIS had been adopted during that year: 


e Students who were severely chronically absent: absent for more than 15 percent of school days in the 2017/18 
school year. 


e Students with disciplinary infractions: suspended or expelled at least once in the 2017/18 school year. 


e Grade 10 students who had not obtained 25 percent of the credits needed to graduate during the 2017/18 
school year, when they were in grade 9.7 


These subgroups of students are not necessarily the same students who were identified and supported by an EIIS. 
An EIIS is supposed to allow districts to identify students rapidly, as soon as they begin to stop meeting attendance, 
behavior, or course progression and academic achievement goals; allow them to continue identifying new 
students as they stop meeting one of these goals; and allow students to move in and out of interventions as 
needed throughout the school year. In contrast, the sample in each of these subgroups is made up of students 
who were identified once (at the end of the year before EIIS adoption). Although students in these subgroups 
might have been more likely to be identified as needing interventions during the program year than the average 
student, they would not be expected to overlap completely with the group of students identified through an EIIS. 
The study team was unable to collect information about which students were identified for and received support 
through an EIIS during the program year. 


For students who were severely chronically absent in the year before EIIS adoption, an EIIS did not change the 
proportion of students who were chronically absent, had disciplinary infractions, met the course-progression goals 
needed to be considered on track to graduate at the end of grade 9, or scored at or above the proficiency level on 
the English language arts test in grade 11 (table C2). The program does appear to have increased by 9.5 percentage 
points the proportion of students in this subgroup with math test scores at or above the proficiency level: the 
comparison districts underperformed their baseline trend’s projection by 4.2 percentage points, while the EIIS 
districts outperformed their baseline trend’s projection by 5.2 percentage points. However, this finding needs to 
be interpreted with caution. Of the 39 impact estimates reported for these subgroups, this is the only one that is 
statistically significant at the 5 percent level. Typically, one would expect to see 1 of every 20 estimates be 
statistically significant at this level in the absence of true effects. 


1 Because the on-track indicator is available only for students in grade 9, this subgroup includes only students in grade 10 whose on-track 
indicators from the previous school year were not missing. 
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Table C2. Estimated effects of early indicator and intervention systems (EIISs) on chronic absenteeism, 
disciplinary infractions, course progression, and academic achievement in 2018/19 for students identified as 
chronically absent in the previous year (2017/18) 
Difference between 
actual and projected 
outcomes 
(percentage points) 


95% confidence interval 


ercentage points 
Estimated (P eeipolnts) 


EIIS Comparison effect” 


p-value for 
Siclalerlae| estimated 
Effect size Janelg effect 


districts 
(se) 


districts? 
(n = 46) 


(percentage Lower Upper 
rexeyfaiasy) bound exo Tare] 


Percentage of students who were chronically absent 


All grades —6.2* 3.8 2.5 —9.9 4.9 —-0.05 3.76 512 
Grade 9 -8.3* -3.5 -4.7 —16.2 6.7 -0.11 5.80 413 
Grade 10 -1.5 -0.3 -1.1 -12.4 10.1 —0.02 5.72 .844 
Grade 11 —-6.2 -8.2* 2.0 —7.1 11.1 0.04 4.63 .665 
Grade 12 -5.4 —4.7 -0.6 —10.8 9.5 -0.01 5.17 .901 
Percentage of students who were severely chronically absent 

All grades —6.6* -4.4 —2.2 -10.7 6.4 —0.06 4.33 .616 
Grade 9 4.4 —10.0 5.7 -8.6 19.9 0.17 7.23 434 
Grade 10 1.8 -1.5 3.4 —9.8 16.5 0.09 6.68 .614 
Grade 11 —-10.7* -9.0 -1.7 —16.6 13.2 —0.04 7.57 .824 
Grade 12 -6.7 3.6 —10.3 -22.1 1.5 —0.24 6.00 .086 
Percentage of students with disciplinary infractions 

All grades 0.4 -0.7 11 3.3 5.5 0.05 2.22 .628 
Grade 9 0.6 -8.1* 8.6 -1.2 18.5 0.33 4.99 .085 
Grade 10 0.4 4.3 -3.9 -14.0 6.2 —0.16 5.13 445 
Grade 11 0.2 3.1 —2.8 —9.8 4.1 -0.13 3.54 425 
Grade 12 -1.3 —2.5 1.2 -3.9 6.2 0.07 2.58 .656 
Percentage of students meeting course progression goals 

Grade 9 4.6 5.8 =1.2 —15.7 13.2 —0.03 7.36 .865 
Percentage of students proficient or above in math 

Grade 11 5.2 —4.2 9.5* 0.3 18.6 0.20 4.63 .042 
Percentage of students proficient or above in English language arts 

Grade 11 -5.3 —6.2 0.9 -15.2 17.0 0.02 8.16 .912 


* Significant at p < .05. 

Note: All values were aggregated to the district level. The chronically absent subgroup included students who were absent for more than 15 percent of 
school days in the previous year. Only districts with at least one student identified for this subgroup in the program year (2018/19) and the last baseline year 
(2017/18) are included in the analysis. A two-tailed t-test was applied to each estimated difference between the EIIS districts and the comparison districts. 
Rounding might cause slight discrepancies in calculating the sums and differences. 

a. Values are the estimated differences between actual and projected outcomes for districts that adopted an EIIS in 2018/19. 

b. Values are the differences between the EIIS districts and the comparison districts in the differences between their actual and projected outcomes. 
Source: Authors’ analysis based on student-level data from the Oregon Department of Education and school-level data from U.S. Department of Education 
(n.d.), for school years 2013/14—2018/19. 


For students who were ever suspended or expelled during the year before (2017/18), the EIS adoption (2018/19), 
an EIIS does not appear to have produced any effects on the measured outcomes (table C3). Some estimated 
effects were large in magnitude, but the small sample sizes mean that it is more likely to find spurious results and 
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less likely for outcomes to be statistically significant. For example, the estimated effect for this subgroup on the 
proportion of students with disciplinary infractions in grade 10 is 9.9 percentage points, but it is not statistically 
significant. 


Table C3. Estimated effects of early indicator and intervention systems (EIISs) on chronic absenteeism, 
disciplinary infractions, course progression, and academic achievement in 2018/19 for students with 
disciplinary infractions in the previous year (2017/18) 
Difference between 
actual and projected 
outcomes 95% confidence interval 
(percentage points) (percentage points) 


Estimated 
EIS Comparison effect? p-value for 
districts? districts (percentage Lower Upper Siilalel-lae} estimated 

Outcome (710) (Ae) points) jefe} Ul aye] bound Effect size lagelg effect 


Percentage of students who were chronically absent 


All grades —5.4* —8.5*** 3.1 —3.7 9.9 0.07 3.47 .372 
Grade 9 -0.5 —2.8 2.3 —10.7 15.3 0.05 6.60 727 
Grade 10 4.3 7.3 3.1 —10.0 16.1 0.07 6.62 .644 
Grade 11 5.5 —13.1* 7.6 —-6.7 21.8 0.16 7.24 .296 
Grade 12 5.8 —11.0* 5.2 8.6 19.0 0.11 7.01 459 
Percentage of students who were severely chronically absent 

All grades —6.0* —6.0* 0.0 -6.6 6.6 0.00 3.36 1.000 
Grade 9 —0.2 —4.3 4.2 7.5 15.8 0.12 5.90 482 
Grade 10 -7.7 -2.4 5.3 -17.4 6.8 —0.14 6.14 391 
Grade 11 —11.0* —9.7 -1.4 -15.1 12.3 —0.03 6.96 .842 
Grade 12 —2.9 7.8 4.9 —9.0 18.8 0.11 7.04 488 
Percentage of students with disciplinary infractions 

All grades 0.5 —4.6* 5.1 -0.4 10.6 0.23 2.79 .067 
Grade 9 —2.4 -5.7 3.3 -7.4 14.0 0.13 5.43 544 
Grade 10 5.9 —4.0 9.9 -0.1 19.8 0.40 5.06 052 
Grade 11 -1.3 -3.0 1.6 8.1 11.4 0.08 4.96 743 
Grade 12 -5.5 -3.3 —2.2 —11.8 7.3 —0.14 4.85 .646 


Percentage of students meeting course-progression goals 

Grade 9 2.8 —2.8 5.6 -6.6 17.8 0.15 6.19 365 
Percentage of students proficient or above in math 

Grade 11 -2.1 2.3 4.4 —13.0 4.2 —0.09 4.36 313 
Percentage of students proficient or above in English language arts 

Grade 11 4.6 -1.3 -3.3 -16.6 9.9 —0.07 6.73 .620 


* Significant at p < .05; *** significant at p < .001. 

Note: All values were aggregated to the district level. The subgroup of students with disciplinary infractions included students who were ever suspended or 
expelled in the previous year. Only districts with at least one student identified for this subgroup in the program year (2018/19) and the last baseline year 
(2017/18) are included in the analysis. A two-tailed t-test was applied to each estimated difference between the EIIS districts and the comparison districts. 
Rounding might cause slight discrepancies in calculating the sums and differences. 

a. Values are the estimated differences between actual and projected outcomes for districts that adopted an EIIS in 2018/19. Rounding might cause slight 
discrepancies in calculating the sums and differences. 

b. Values are the differences between the EIIS districts and the comparison districts in the differences between their actual and projected outcomes. 
Source: Authors’ analysis based on student-level data from the Oregon Department of Education and school-level data from U.S. Department of Education 
(n.d.), for school years 2013/14—2018/19. 


For students in grade 10 who did not attain the credits needed to be considered on track to graduate at the end 
of grade 9 the study team was able to assess only the effects of an EIIS on three outcome measures: the two 
chronic absenteeism measures and the disciplinary infractions measure. The program does not appear to have 
produced any effects on these three outcomes (table C4). 


Table C4. Estimated effects of early indicator and intervention systems (EIISs) on chronic and severely chronic 

absenteeism and disciplinary infractions in 2018/19 for grade 10 students who had not obtained 25 percent of 

the credits needed to graduate by the end of grade 9 in the previous year (2017/18) 
Difference between 


actual and projected 
outcomes 95% confidence interval 


EIIS Comparison p-value for 
districts? districts” Estimated Lower Upper Siilalel-lae} estimated 
(7010) (ees -)) effect‘ leYol Ur are] bound Effect size lagele effect 


Percentage of students who were chronically absent 

Grade 10 —1.0 —2.5 1.5 —9.0 11.9 0.03 5.31 0.780 
Percentage of students who were severely chronically absent 

Grade 10 -3.7 1.2 4.9 —14.8 4.9 -0.13 5.02 0.326 
Percentage of students with disciplinary infractions 

Grade 10 2.4 3.4 -1.0 -6.1 4.1 —0.04 2.61 0.700 


Note: All values were aggregated to the district level. The subgroup of students who failed to meet the course-progression goal included grade 10 students 
who failed to obtain at least 25 percent of the credits required for high school graduation by the end of grade 9 in the previous year. Only districts with at 
least one student identified for this subgroup in the program year (2018/19) and the last baseline year (2017/18) were included in the analysis. A two-tailed 
t-test was applied to each estimated difference between the EIIS districts and the comparison districts. None of the findings was statistically significant. 
Rounding might cause slight discrepancies in calculating the sums and differences. 

Values are the estimated differences between actual and projected outcomes for districts that adopted an EIIS in 2018/19. Rounding might cause slight 
discrepancies in calculating the sums and differences. 

b. Values are the estimated differences between the actual and projected outcomes for the comparison districts. 

c. Values are the differences between the EIIS districts and the comparison districts in the differences between their actual and projected outcomes. 
Source: Authors’ analysis based on student-level data from the Oregon Department of Education and school-level data from U.S. Department of Education 
(n.d.), for school years 2013/14—2018/19. 


Overall, there was no strong evidence indicating that an EIIS benefited subgroups of students not meeting 
attendance, behavior, and course-progression goals in the year before the EIIS adoption. 
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